test(perf): Add Llama-3_1-Nemotron-Ultra-253B-v1 perf tests (cpp)
#4446
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
venkywonka
requested review from
LarryXFly,
Copilot,
kaiyux,
ruodil,
schetlur-nv and
tijyojwad
and removed request for
Copilot and
ruodil
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces performance tests for the Llama-3_1-Nemotron-Ultra-253B-v1 model using the cpp TRT backend to ensure that both low- and high-concurrency scenarios pass within CI limits.
- Added new test entries with appropriate parameters (max batch size, input/output lengths, concurrency, etc.) in the QA test list.
- Updated model mapping in test_perf.py to include the new ultra model for both native and Hugging Face identifiers, and appended a build flag when remote code is trusted.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/integration/test_lists/qa/trt_llm_release_perf_test.yml | Added new performance test entries for Llama-3_1-Nemotron-Ultra-253B-v1. |
| tests/integration/defs/perf/test_perf.py | Added new model mapping entries and introduced a build flag for TRUST_REMOTE_CODE_MODELS. |
venkywonka
requested review from
LarryXFly,
Copilot,
kaiyux,
ruodil,
schetlur-nv and
tijyojwad
and removed request for
LarryXFly,
kaiyux,
schetlur-nv and
tijyojwad
Contributor
There was a problem hiding this comment.
Pull Request Overview
Adds C++-backend FP8 performance tests for the llama-3.1-nemotron-ultra-253b-v1 model and hooks it into the test runner, including enabling remote code trust for quantized builds
- New YAML entries in
trt_llm_release_perf_test.ymlfor low/high concurrency FP8 benchmarks - Model mapping definitions added in
test_perf.pyfor both C++ and HF backends - Auto-enables
--trust_remote_codefor models listed inTRUST_REMOTE_CODE_MODELSduring build
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/integration/test_lists/qa/trt_llm_release_perf_test.yml | Added four new perf test invocations covering C1–C4 scenarios for the new Ultra-253B model |
| tests/integration/defs/perf/test_perf.py | Registered llama_v3.1_nemotron_ultra_253b[_hf] mappings and added --trust_remote_code flag |
Comments suppressed due to low confidence (3)
tests/integration/defs/perf/test_perf.py:58
- The repository path uses 'Llama-3_1...' with an underscore instead of the dot notation ('Llama-3.1...'), which is inconsistent with other model paths and may break resolution. Update to match the existing naming convention.
"nemotron-nas/Llama-3_1-Nemotron-Ultra-253B-v1",
tests/integration/defs/perf/test_perf.py:105
- The HuggingFace model path uses an underscore in 'Llama-3_1...' instead of 'Llama-3.1...'; this diverges from established naming and may cause lookup failures. Please correct it.
"nvidia/Llama-3_1-Nemotron-Ultra-253B-v1",
tests/integration/defs/perf/test_perf.py:932
- Automatically setting
--trust_remote_code=Truecan pose security risks if unreviewed code is pulled. Ensure these models are audited or document why remote code trust is safe here.
if self._config.model_name in TRUST_REMOTE_CODE_MODELS:
acbc33c to
02e6b02
Compare
tijyojwad
approved these changes
May 19, 2025
LarryXFly
approved these changes
May 21, 2025
ruodil
approved these changes
May 21, 2025
Signed-off-by: Venky Ganesh <[email protected]>
venkywonka
force-pushed
the
user/venky/ll-nemo-ultra-perf-tests-cpp
branch
from
eb62a33 to
0bb8c75
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #6157 [ run ] triggered by Bot |
Collaborator
|
PR_Github #6157 [ run ] completed with state |
venkywonka
added a commit
to venkywonka/TensorRT-LLM
that referenced
this pull request
May 22, 2025
…VIDIA#4446) ultra Signed-off-by: Venky Ganesh <[email protected]>
chzblych
pushed a commit
that referenced
this pull request
May 28, 2025
…ests (cpp) (#4446) (#4590) Signed-off-by: Venky Ganesh <[email protected]>
darraghdog
pushed a commit
to darraghdog/TensorRT-LLM
that referenced
this pull request
Jun 3, 2025
…VIDIA#4446) ultra Signed-off-by: Venky Ganesh <[email protected]> Signed-off-by: darraghdog <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add
llama-3.1-nemotron-ultra-253b-v1perf tests converage.This only adds cpp backend for fp8.
This is because, since a model this big rarely is used in bf16 precision, only resource-efficient to test on fp8. PyT backend fp8 requires pre-quantized fp8 checkpoints (we currently don't have it added).
Invariants
max_batch_sizetrtllm-benchFour sequence profiles were benchmarked:
reqs = 8,con = 1reqs = 250,con = 250Performance Summary
(tok/s)
(ms)
Latency percentiles
Concurrency = 1 (C1 & C2)
Concurrency = 250 (C3 & C4)